Robots.txt is a text file that can be placed on a server, and the file basically tells spiderbots (from the Internet Wayback Machine, search engines like [[Jewgle]], and website archiving software) to fuck off.
In simpler terms, if this file were to be placed on ED's server, Google and other search engines wouldn't be able to archive copies of webpages, the Wayback Machine wouldn't be able to either, and you wouldn't be able to use some webpage archiving program to do it yourself.
This file is usually put on websites who don't want to have to shit themselves because some faggot posted something embarrassing, and they don't want to leave an internet record, so they usually can take down the page and put it back up minus the incriminating/embarrassing shit.
However, this does not stop you from taking screencaps, so take those if you find a website with one of these.
If you don't know if a website has one of these, go to Google, find the webpage, and click on the "Cached" link under the page link, and if it tells you something like "this page cannot be archived due to Robots.txt", then you'll have to save copies of the webpage by hand or take screencaps. You can also find out if it has this text file by punching in the address into the [[Archive.org|Internet Wayback Machine]], and it will tell you the same thing if the server has this file.
The present day version of robots.txt is cloudfare or literally any website that gives a verification pop up before you can see it, because the spider bots obvoiusly can not bypass the pop ups, and thus, they will fail to actually archive the page.